Open source, standard tooling for experimental protocols: towards Registered reports
Old school:
What if… → Garden of forking paths (Rubin 2017)
p-hacking, hypothesizing after the results are known (i.e. HARKing)
The need for significance and novelty
False positives research
Open-Science-Collaboration (2015)
Baker (2016)
Improve methods: increase n, better measures and manipulations, improve design, piloting
Reduce temptations: p-hacking, hypothesizing after the results are known (i.e. HARKing), selective reporting → preregistration, internal replications
Openness: transparency of the research process, sharing methods, materials, procedures, and data
Nosek et al. (2022)
RRs were conceived to alter the incentives for authors and journals away from producing novel, positive, clean findings and towards conducting and publishing rigorous research on important questions. Soderberg et al. (2021)
RRs outperformed comparison papers on all 19 criteria (Soderberg et al. 2021)
Sizable improvements in:
Statistically indistinguishable in:
RRs could improve research quality while reducing publication bias…
More open, preregistered, reproducible by default
It does not matter if p value is < 0.05
Less incentives for p-hacking
More trustworthy results
Before having the data available, it is hard to know how to analyze it
There are always surprises when receiving new data. How can I create an analysis plan that will hold?
We (CSCN; ~5-10 PI’s) used different technologies to develop experiments: Psychopy, Qualtrics, Limesurvey, jsPsych, etc.
Each of these has advantages and disadvantages.
Mostly, pragmatic aspects guided the decision: lab history and resources, coding experience, type of experiment (EEG/behavioral, lab/online), …
Each protocol started almost from scratch. A single pre-existing task would define the technology used.
Multiple implementations of the same tasks in different technologies, not always exact replicas, not always easy to find.
Some would work in certain computers, other did not.
Output data wildly different, data preparation a hard task and error prone.
2 questions voluntary survey:
Open source tools to help create experimental paradigms with jsPsych, simulate participants and standardize the data preparation and analysis
1) jsPsychMaker
A big catalog of tasks in jsPsychMaker. Each task runs with jsPsychMonkeys to create virtual participants, and have a script in jsPsychHelpeR to automate data preparation (re-coding, reversing items, calculating dimensions, etc.).
The final goal is to help you have the data preparation and analysis ready before collecting any real data, reducing errors in the protocols, and making the move towards registered reports easier.
Current developers:
Initial idea and development:
Discussions, ideas, testing:
Creating new tasks is as simple as:
Copy paste items, adapt csv/excel file
Create protocol (see next slide)
Create a protocol with three existing tasks plus the adapted Example tasks:
Release a single Monkey and take a look:
Release 10 Monkeys in parallel:
Create project for data preparation:
Create a task correction script:
Visualize and run data preparation:
Create protocol, simulate participants and prepare data…
Let’s try to download the data, process it and show a report with the results:
Plan A: run Experiment Issues project
Plan B: If something fails, we always have the monkeys!
Very easy to create new scales, and simple tasks, but complex experimental tasks require javascript and HTML knowledge (although there are a good number of examples available)
Data preparation for new experimental tasks requires expertise in R (simple surveys not so much)
Analysis reports require some R knowledge (simple templates available)
Requires access to a server for online tasks
Only behavioral tasks
Javascript programmers
R programmers
Task creators
Testers
With jsPsychR, protocols are standardized with (mostly) clean code, open source, and based on web standards
Data preparation is 90% automatic, standardized, and beautiful
Less errors in protocols and in data preparation
Time from idea to protocol much lower
When errors are found and fixed, old protocols can benefit from the corrections, old results can be checked, …
Trivial to work on analysis before collecting human data
Much easier to write up a good analysis plan, share it, improve it, …
Create fully reproducible papers and results’ reports easily
Sharing protocol, materials, data preparation is trivial (single command)
Creating future-proof fully reproducible data preparation projects (with Docker) is one command away
For more information about RRs, templates, checklists, list of participating journals (>300), etc.:
Also, check out the future:
And our manual:
Gorka Navarrete
gorkang@gmail.com
https://fosstodon.org/@gorkang